Impact of predictor measurement heterogeneity across settings on performance of prediction models: a measurement error perspective

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out-of-sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between deriviation and validation data is common, the impact on the out-of-sample performance is not well studied. Using analytical and simulation approaches, we examined out-of-sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.Comment: 32 pages, 4 figure

Quality of reporting of drug exposure in pharmacoepidemiological studies

Publisher's version (útgefin grein)Purpose: Exposure definitions vary across pharmacoepidemiological studies. Therefore, transparent reporting of exposure definitions is important for interpretation of published study results. We aimed to assess the quality of reporting of exposure to identify where improvement may be needed. Method: We systematically reviewed observational pharmacoepidemiological studies that used routinely collected health data, published in 2017 in six pharmacoepidemiological journals. Reporting of exposure was scored using 11 items of the ISPE-ISPOR guideline on reporting of pharmacoepidemiological studies. Results: Of the 91 studies included, all studies reported the type of exposure (100%), while most reported the exposure risk window (85%) and the exposure assessment window (98%). Operationalization of the exposure window was described infrequently: 16% (14/90) of the studies explicitly reported the presence or absence of an induction period if applicable, 11% (5/47), and 35% (17/49) reported how stockpiling and gaps between exposure episodes were handled, respectively, and 35% (17/49) explicitly mentioned the exposure extension. Switching/add-on was reported in 62% (50/81). How switching between drugs was dealt with and specific drug codes were reported in 52 (57%) and 24 (26%) studies, respectively. Conclusion: Publications of pharmacoepidemiological studies frequently reported the type of exposure, the exposure risk window, and the exposure assessment window. However, more details on exposure assessment are needed, especially when it concerns the operationalization of the exposure risk window (eg, the presence or absence of an induction period or exposure extension, handling of stockpiling and gaps, and specific codes), to allow for correct interpretation, reproducibility, and assessment of validity.RHHG was funded by the Netherlands Organization for Scientific Research (ZonMW‐Vidi project 917.16.430) and an LUMC fellowship.Peer Reviewe

Tell me what you want, what you really really want: Estimands in observational pharmacoepidemiologic comparative effectiveness and safety studies

PURPOSE: Ideally, the objectives of a pharmacoepidemiologic comparative effectiveness or safety study should dictate its design and data analysis. This paper discusses how defining an estimand is instrumental to this process. METHODS: We applied the ICH-E9 (Statistical Principles for Clinical Trials) R1 addendum on estimands - which originally focused on randomized trials - to three examples of observational pharmacoepidemiologic comparative effectiveness and safety studies. Five key elements specify the estimand: the population, contrasted treatments, endpoint, intercurrent events, and population-level summary measure. RESULTS: Different estimands were defined for case studies representing three types of pharmacological treatments: (1) single-dose treatments using a case study about the effect of influenza vaccination versus no vaccination on mortality risk in an adult population of ≥60 years of age; (2) sustained-treatments using a case study about the effect of dipeptidyl peptidase 4 inhibitor versus glucagon-like peptide-1 agonist on hypoglycemia risk in treatment of uncontrolled diabetes; and (3) as needed treatments using a case study on the effect of nitroglycerin spray as-needed versus no nitroglycerin on syncope risk in treatment of stabile angina pectoris. CONCLUSIONS: The case studies illustrated that a seemingly clear research question can still be open to multiple interpretations. Defining an estimand ensures that the study targets a treatment effect that aligns with the treatment decision the study aims to inform. Estimand definitions further help to inform choices regarding study design and data-analysis and clarify how to interpret study findings

Clinical decision support for ExtraCorporeal Membrane Oxygenation:Will we fly by wire?

Prognostic modelling techniques have rapidly evolved over the past decade and may greatly benefit patients supported with ExtraCorporeal Membrane Oxygenation (ECMO). Epidemiological and computational physiological approaches aim to provide more accurate predictive assessments of ECMO-related risks and benefits. Implementation of these approaches may produce predictive tools that can improve complex clinical decisions surrounding ECMO allocation and management. This Review describes current applications of prognostic models and elaborates on upcoming directions for their clinical applicability in decision support tools directed at improved allocation and management of ECMO patients. The discussion of these new developments in the field will culminate in a futuristic perspective leaving ourselves and the readers wondering whether we may “fly ECMO by wire” someday.</p

Changing predictor measurement procedures affected the performance of prediction models in clinical examples

Objectives: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols. Study Design and Setting: We examined the effects of various scenarios of predictor measurement heterogeneity in r

Prediction meets causal inference: the role of treatment in clinical prediction models

In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a ‘predictimand’ framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference

Prediction models for diagnosis and prognosis of covid-19: : systematic review and critical appraisal

Readers’ note This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity. Funding: LW, BVC, LH, and MDV acknowledge specific funding for this work from Internal Funds KU Leuven, KOOR, and the COVID-19 Fund. LW is a postdoctoral fellow of Research Foundation-Flanders (FWO) and receives support from ZonMw (grant 10430012010001). BVC received support from FWO (grant G0B4716N) and Internal Funds KU Leuven (grant C24/15/037). TPAD acknowledges financial support from the Netherlands Organisation for Health Research and Development (grant 91617050). VMTdJ was supported by the European Union Horizon 2020 Research and Innovation Programme under ReCoDID grant agreement 825746. KGMM and JAAD acknowledge financial support from Cochrane Collaboration (SMF 2018). KIES is funded by the National Institute for Health Research (NIHR) School for Primary Care Research. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. GSC was supported by the NIHR Biomedical Research Centre, Oxford, and Cancer Research UK (programme grant C49297/A27294). JM was supported by the Cancer Research UK (programme grant C49297/A27294). PD was supported by the NIHR Biomedical Research Centre, Oxford. MOH is supported by the National Heart, Lung, and Blood Institute of the United States National Institutes of Health (grant R00 HL141678). ICCvDH and BCTvB received funding from Euregio Meuse-Rhine (grant Covid Data Platform (coDaP) interref EMR187). The funders played no role in study design, data collection, data analysis, data interpretation, or reporting.Peer reviewedPublisher PD

New-user and prevalent-user designs and the definition of study time origin in pharmacoepidemiology: a review of reporting practices

Background: Guidance reports for observational comparative effectiveness and drug safety research recommend implementing a new-user design whenever possible, since it reduces the risk of selection bias in exposure effect estimation compared to a prevalent-user design. The uptake of this guidance has not been studied extensively. Methods: We reviewed 89 observational effectiveness and safety cohort studies published in six pharmacoepidemiological journals in 2018 and 2019. We developed an extraction tool to assess how frequently new-user and prevalent-user designs were reported to be implemented. For studies that implemented a new-user design in both treatment arms, we extracted information about the extent to which the moment of meeting eligibility criteria, treatment initiation, and start of follow-up were reported to be aligned. Results: Of the 89 studies included, 40% reported implementing a new-user design for both the study exposure arm and the comparator arm, while 13% reported implementing a prevalent-user design in both arms. The moment of meeting eligibility criteria, treatment initiation, and start of follow-up were reported to be aligned in both treatment arms in 53% of studies that reported implementing a new-user design. We provided examples of studies that minimized the risk of introducing bias due to unclear definition of time origin in unexposed participants, immortal time, or a time lag. Conclusions: Almost half of the included studies reported implementing a new-user design. Implications of misalignment of study design origin were difficult to assess because it would require explicit reporting of the target estimand in original studies. We recommend that the choice for a particular study time origin is explicitly motivated to enable assessment of validity of the study